Skip to content

Resolve mallinfo2 at runtime; fix always-zero malloc arena counters in el8 builds - #918

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits into
mainfrom
implement-jira-ticket
Nov 12, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits into
mainfrom
implement-jira-ticket

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Resolves mallinfo2 at runtime instead of gating the call on the glibc build headers. mallocInfo.cpp looks the symbol up with dlsym(RTLD_DEFAULT, "mallinfo2") and declares a ten-field size_t mirror of struct mallinfo2; static_asserts pin the mirror to the real struct wherever the headers declare it. The malloc_* arena counters are therefore populated in every release build whenever the runtime glibc provides mallinfo2 (2.33+), and stay unset otherwise.

The counter semantics are stated where they live: updateMallocArenaStats() documents the glibc int_mallinfo meaning of each field (arena sums av->system_mem over all arenas; keepcost is the main arena's top chunk only), and doc/reference/ProfilerMemoryRequirements.md gains a "Malloc Arena Counters" section covering availability and the unset cases.

Motivation:

PROF-16350: the malloc_* arena counters are the base for planned glibc arena-blowup detection, but they were compiled out of shipped builds: mallinfo2() was gated on __GLIBC_MINOR__ >= 33 at compile time, and the release libraries are built in the el8 image against glibc 2.28 headers — so every shipped malloc_arena_bytes/malloc_in_use_bytes was 0. A dlsym string lookup adds no GLIBC_2.33 symbol-version reference, so the same binary picks mallinfo2 up from any runtime glibc that has it, regardless of build headers.

Alternatives rejected:

  • mallinfo() fallback: its int fields truncate past 2 GiB. Counters that silently wrap are worse than unset counters.
  • Publishing zeros on unsupported runtimes: zeros read as "no allocator memory". Unset (the same state as musl today) is the honest signal.

Additional Notes:

  • Agentic review ran over the full change set (sphinx, deep effort): 10 findings — 8 fixed, 2 dismissed. Every post-review delta was re-reviewed (sphinx, standard effort). Two LOW findings remain open, both latent test-quality items in the ldd version probe of MallocArenaCountersTest: a failed probe on a modern-glibc host would assert counters are 0 instead of skipping, and a three-component pre-release banner (2.38.9000) would misprint the version in assertion messages. The 2.33 gate outcome stays correct in both cases. Follow-up ticket on request.

  • Human review — recommended, not required: no critical area from doc/WorkTracking.md is touched (no public API, security, schema, dependency, or signal-handler change; the mallinfo2 lookup runs on the JFR flush path only). Recommended for two reasons: shipped-artifact JFR behavior changes — the malloc_* counters go from always-0 to real values, which consumers of these counters should acknowledge — and the unset-not-zero fallback semantics plus the documented-contract placement are recorded decisions on PROF-16350 worth a second pair of eyes.

How to test the change?:

  • ddprof-test MallocArenaCountersTest (new): non-zero malloc_arena_bytes/malloc_in_use_bytes through the full JFR flush path on a runtime glibc ≥ 2.33; counters unset otherwise (musl, older glibc); sanitizer/replacement-allocator runs skip the non-zero oracle.
  • ddprof-lib mallocArenaStats_ut.cpp (reworked): glibc accounting-identity, hblkhd, and malloc_trim premise tests against the same runtime-resolved path; skips at runtime where mallinfo2 is absent.

Evidence:

Check Result
:ddprof-lib:gtestDebug and :ddprof-lib:gtestRelease_mallocArenaStats, glibc 2.35 host passed
:ddprof-test:testDebug -Ptests=MallocArenaCountersTest passed
el8 container build (glibc 2.28 headers, no mallinfo2 declaration) passed
el8-built library on glibc 2.35: counters non-zero passed
Same library in oraclelinux:7 (glibc 2.17): loads, counters unset, max symbol version GLIBC_2.17 (objdump -T) passed
PR CI, head 929ca356f passed — 124 pass, 9 skip, 0 fail

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16350

@jbachorik jbachorik added the AI label Oct 28, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5133806 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #43267243603 | Commit: ff9a60e | Duration: 21m 8s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-12 17:38:23 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 51338065

@jbachorik jbachorik changed the title Resolve mallinfo2 at runtime; fix always-zero malloc arena counters in el8 builds (PROF-16350) Resolve mallinfo2 at runtime; fix always-zero malloc arena counters in el8 builds Oct 28, 2026
@jbachorik
jbachorik marked this pull request as ready for review October 28, 2026 16:46
@jbachorik
jbachorik requested a review from a team as a code owner October 28, 2026 16:47
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T16:50:32.247727Z c65bb37 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c65bb37f75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The runtime lookup removes the build-header gate without introducing a direct mallinfo2 symbol dependency. Validation was limited to static review and a C++ syntax check; runtime compatibility and JFR behavior were not exercised.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit c65bb37

@jbachorik
jbachorik force-pushed the implement-jira-ticket branch from c65bb37 to a873a90 Compare November 12, 2026 09:43

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The runtime lookup avoids a direct mallinfo2 symbol dependency, and the mirrored layout is pinned against newer glibc headers. Runtime compatibility and JFR integration were not exercised in this review.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit a873a90

@dd-octo-sts

dd-octo-sts Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervmwuk1x
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 21.1.8 (6ubuntu1)
Date:Thu Nov 12 14:59:20 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs2
Logic error
Dereference of null pointer1
Unused code
Dead initialization1

Filter Results by File

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Unused codeDead initializationreferenceChains.cppapplyOomRamp4971
Logic errorDereference of null pointerperfEvents_linux.cppunregisterThread72026

@jbachorik
jbachorik force-pushed the implement-jira-ticket branch from 929ca35 to efb2a4e Compare November 12, 2026 11:20

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Runtime symbol resolution avoids a GLIBC_2.33 link-time dependency while preserving unset counters when mallinfo2 is unavailable. The completed static review identified no actionable defect.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit efb2a4e

@zhengyu123

Copy link
Copy Markdown
Contributor

Review findings (static review of the diff)

This review read the diff only. It did not build or run anything. Please verify each point before acting.

Correctness and robustness

  1. mallocInfo.cpp:175 – the mallinfo2 mirror struct is not compiler-checked on el8.
    mallinfo2 is called through a hand-written function-pointer type and a mirror struct. The static_asserts that pin the mirror to glibc's struct sit inside #if __GLIBC_PREREQ(2,33). They do not compile on the el8 release image, which is the build this code targets. If glibc extends struct mallinfo2, the el8 binary would use a wrong layout. For a struct returned by value, this can mean a wrong return-buffer size and a stack overwrite.

  2. mallocInfo.cpp:179 – dlsym(RTLD_DEFAULT, "mallinfo2") may not resolve to glibc.
    It returns the first match in global scope. A preloaded allocator can export its own mallinfo2 with a different layout. glibcMallocActive() only probes mallctl and tc_malloc_size, so it can miss such an allocator. The profiler would then read the result with glibc's layout and publish wrong values.

  3. mallocInfo.cpp:140 – the pinning checks depend on include order.
    #include <malloc.h>, <stddef.h> and the static_asserts sit mid-file inside nested conditions. __GLIBC__ is defined only after a libc header is included. If <dlfcn.h> stops pulling in features.h, the guard turns false and the checks are skipped without a diagnostic.

Efficiency

  1. flightRecorder.cpp:2161 – dlsym on every JFR chunk flush.
    Each flush does one dlsym lookup, and glibcMallocActive() adds two more. That is three loader-lock acquisitions per flush. The pointer does not change after startup. A function-local static or std::call_once would resolve it once.

Tests

  1. MallocArenaCountersTest.java:685 – ldd --version can report the wrong glibc.
    ldd reports the system default glibc, not necessarily the one the JVM loaded. In a container or chroot, or with a bundled sysroot, the test can assert the wrong expectation. If ldd is missing, ProcessBuilder.start throws IOException and the test errors instead of skipping.

  2. mallocArenaStats_ut.cpp:256 – new #error on unknown libc.
    A Linux gtest build that defines neither __GLIBC__ nor -D__musl__ now fails to compile. Before, those tests were compiled out. This matters for any build that does not go through GtestTaskBuilder.

  3. mallocArenaStats_ut.cpp:284 – MmappedChunkIsReportedInHblkhdOnly is fragile.

    • It mallocs 256 MiB and expects an mmap-served chunk. Under RLIMIT_AS, a cgroup limit or disabled overcommit, the malloc fails and the test skips, which can hide regressions.
    • Other gtest threads or a lowered mmap threshold can break the uordblks/arena deltas.
    • The exact-equality assert arena == uordblks + fordblks compares separate snapshots with no stop-the-world.

@jbachorik

Copy link
Copy Markdown
Collaborator Author

@zhengyu123 Thanks — verified each point against the code. Three were real and are fixed in 64a395a / 5133806; the rest I believe are covered by the code as written, with the reasoning below.

Fixed:

  1. Include order — mallocInfo.cpp now includes <features.h> explicitly on Linux, before the guard, so the pins no longer depend on <dlfcn.h> transitively defining __GLIBC__/__GLIBC_PREREQ.
  2. dlsym per flush — both mallocArenaSnapshot() and glibcMallocActive() now resolve once into zero-initialized statics, published with acquire/release (__atomic_*, matching the repo's arm64 ordering policy). No loader-lock acquisition per flush; a racing first caller can only repeat the same lookup.
  3. ldd probe — runtimeGlibcVersion() no longer throws on probe failure: missing ldd, timeout, interruption, and unparsable banners all skip via a JUnit assumption instead of erroring or asserting the wrong branch. The trailing-token regex now requires a token boundary, so a 2.38.9000-style banner is inconclusive rather than mis-parsed. The system-glibc assumption (ldd's glibc == the libc the JVM loaded) holds in every supported test image and is documented in the javadoc.

Not changed, with reasoning:

  1. Mirror unchecked on el8 — on el8 headers struct mallinfo2 does not exist, so there is nothing to pin against; the mirror is the definition there. The residual risk is glibc growing the struct under the same symbol, which would break the return-by-value ABI of every binary compiled against 2.33+ headers — glibc's ABI contract (new fields come with new symbols, as mallinfo→mallinfo2 did) is the actual protection, and it is the same contract every pre-2.33-compiled consumer relies on. All builds whose headers do declare the struct compile the field pins; the el8 path is cross-checked at runtime by the accounting-identity/hblkhd premise tests. This is documented in mallocInfo.cpp.
  2. RTLD_DEFAULT may hit another library's mallinfo2 — exporting a symbol named mallinfo2 is itself a declaration of glibc-mallinfo2 ABI compatibility; a library shipping a different layout under that name is broken by contract. The two real-world replacement allocators (tcmalloc, jemalloc) are excluded by glibcMallocActive() and covered by the LD_PRELOAD gtest. Fetching libc's own symbol specifically would need dl_iterate_phdr + dlopen on the flush path — not worth it for advisory counters absent a concrete allocator that both evades the probes and exports the name.
  3. #error on unknown libc — intentional tripwire, documented at the site: a Linux build that can classify neither glibc nor musl is misconfigured, and the silent alternative is trimArenas() compiling to a no-op with the trim premise tests vanishing. The error message names the fix (-D__musl__).
  4. MmappedChunkIsReportedInHblkhdOnly — the 256 MiB request is always above the dynamic mmap-threshold cap (32 MiB on LP64), so it is always mmap-served; verified empirically on glibc 2.35 and 2.41 in both a fresh process and the full-suite sequence. The gtest binaries are single-threaded, and expectMallinfoInvariants checks a single snapshot, not deltas across snapshots. The malloc-failure path skips rather than fails deliberately: in an RLIMIT_AS/cgroup-constrained environment a hard fail would be routine flakiness; the skip prints its reason so it is visible in review.

The remaining mutation-adequacy note (sentinel comparisons in glibcMallocActive() are untested) is covered by the existing GlibcMallocActive.ReportsActiveWithNoReplacementAllocatorLoaded / ReportsInactiveUnderSanitizerAllocator gtests, which call Recording::glibcMallocActive() via RecordingTestAccessor and assert both verdict directions.

This review also caught a real bug I had introduced while fixing an earlier one (the first caching version used plain non-atomic statics — formally UB on concurrent flushes); fixed with acquire/release publication in 5133806.

@rkennke rkennke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thank you!

@jbachorik

Copy link
Copy Markdown
Collaborator Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-12 15:43:49 UTC ℹ️ Start processing command /merge


2026-11-12 15:43:54 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 53m (p90).


2026-11-12 16:42:59 UTC ℹ️ MergeQueue: Readding this merge request to the queue because another merge request processed with yours failed. No action is needed from your side.


2026-11-12 17:14:08 UTC ℹ️ MergeQueue: This merge request was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants